Bayesian Model Averaging for Improving Performance of the Naïve Bayes Classifier

Feature selection has proved to be an effective way to reduce the model complexity while giving a relatively desirable accuracy, especially, when data is scarce or the acquisition of some feature is expensive. However, the single selected model may not always generalize well for unseen test data whereas other models may perform better. Bayesian Model Averaging (BMA) is a widely used approach to address this problem and there are several methods for implementing BMA. In this project, a formula appropriate for applying BMA on Naive Bayesian classifier is derived and the corresponding algorithm is also developed. We expect this modified classifier can always do better than Naive Bayes classifier and is competitive with other linear classifiers such as SVM and Logistic Regression, so a series of performance comparison on several different classifiers are made. To provide a comprehensive evaluation, we build all of the test classifiers under same framework API, use UCI data sets as data sources to train classifiers and apply cross-validation to assess statistical significance of the results. The results indicate three property of our modified classifier. Firstly, the prediction accuracy of our modified classifier is either better than Naive Bayes or, at least, equal to that of Naive Bayes and, in some cases, even better than SVM and Logistic Regression. Secondly, the running speed is faster than SVM and Logistic Regression. Thirdly, time complexity of the modified classifier is linear with respect to number of data as well as number of features. Keywords— Bayesian Model Averaging, Naive Bayes, Feature Selection.